Network-Aware Locality Scheduling for Distributed Data Operators in Data Centers

Large data centers are currently the mainstream infrastructures for big processing. As one of most fundamental tasks in these environments, efficient execution distributed operators (e.g., join and aggregation) still challenging current systems, key performance issues is network communication time. State-of-the-art methods trying to improve that problem focus on either application-layer locality optimization reduce traffic or network-layer flow increase bandwidth utilization. However, techniques two layers totally independent from each other, gains a joint perspective have not yet been explored. In this article, we propose novel approach called NEAL (NEtwork-Aware Locality scheduling) bridge gap, consequently further time operators. We present detailed design implementation NEAL, our experimental results demonstrate always performs better than approaches different workloads configurations.